Gradient Flow in Sparse Neural Networks and How Lottery Tickets Win

نویسندگان

چکیده

Sparse Neural Networks (NNs) can match the generalization of dense NNs using a fraction compute/storage for inference, and have potential to enable efficient training. However, naively training unstructured sparse from random initialization results in significantly worse generalization, with notable exceptions Lottery Tickets (LTs) Dynamic Training (DST). In this work, we attempt answer: (1) why networks performs poorly and; (2) what makes LTs DST exceptions? We show that poor gradient flow at propose modified connectivity. Furthermore, find methods improve during over traditional methods. Finally, do not flow, rather their success lies re-learning pruning solution they are derived — however, comes cost learning novel solutions.

برای دانلود رایگان متن کامل این مقاله و بیش از 32 میلیون مقاله دیگر ابتدا ثبت نام کنید

اگر عضو سایت هستید لطفا وارد حساب کاربری خود شوید

منابع مشابه

How To Win The Doctor Lottery.

Not every doctor-patient encounter is healing, and it can seem a game of chance. One patient explores what it takes to win.

متن کامل

Electronic Lottery Tickets as Micropayments

We present a new micropayment scheme based on the use of “electronic lottery tickets.” This scheme is exceptionally efficient since the bank handles only winning tickets, instead of handling each micro-

متن کامل

Tickets and Currencies Revisited: Extensions to Multi-Resource Lottery Scheduling

Lottery scheduling’s ticket and currency abstractions provide a resource management framework that allows for both flexible allocation and insulation between groups of processes. We propose extensions to this framework that enable greater flexibility while preserving the ability to isolate groups of processes. In particular, we present a mechanism that allows processes to modify their own resou...

متن کامل

Tickets and Currencies Revisited: Extensions to Multi-Resouce Lottery Scheduling

Lottery scheduling’s ticket and currency abstractions provide a resource management framework that allows for both flexible allocation and insulation between groups of processes. We propose extensions to this framework that enable greater flexibility while preserving the ability to isolate groups of processes. In particular, we present a mechanism that allows processes to modify their own resou...

متن کامل

ذخیره در منابع من


  با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید

ژورنال

عنوان ژورنال: Proceedings of the ... AAAI Conference on Artificial Intelligence

سال: 2022

ISSN: ['2159-5399', '2374-3468']

DOI: https://doi.org/10.1609/aaai.v36i6.20611